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Box PCT 

Assistant Commissioner for Patents 
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ATTENTION: EO/US 

NOTE. The completion of those filing requirements that can be made at a time later than 30 months from the priority date 
results from the Commissioner exercising his judgment under the authority granted under 35 USC 371(d). The filing 
receipt mil show the actual date of receipt of the last item completing the entry into the national phase. See 3 7 C.F.R. 
§1.491 which states: "An international application enters the national state when the applicant has filed the 
i documents and fees required by 35 USC 371(c) within the periods set forth in § 1.494 and § 1,495. " 



CERTIFICATION UNDER 37 C.F.R. 1.10* 

(Express Mail label number is mandatory.) 
(Express Mail certification is optional.) 

I hereby certify that this correspondence and the documents referred to as attached therein are being deposited with the 
United States Postal Service on this date July 24, 2001 , in an envelope as "Express Mail Post Office to Addressee," Mailing 
Label Number EL7282141 15US , addressed to the: Assistant Commissioner for Patents, Washington, D.C. 20231, 



JENNIFER RASHKIN 



?pe or print name of person mailing paper) 




WARNING: Certificate of mailing (first class) or facsi/mjte transmission procedures of 37 CF.R. 1.8 cannot be used 

to obtain a date of mailing or transmissibn for this correspondence. 

* WARNING: Each paper or fee filed by "Express Mail" must have the number of the "Express Mail" mailing label 
placed thereon prior to mailing. 37 C.F.R. 1.10(b). 

"Since the filing of correspondence under §1.10 without the Express Mail mailing label thereon is an 
oversight that can be avoided by ike exercise of reasonable care, requests for waiver of this requirement 
will not be granted on petition. " Notice of Oct. 24, 1996, 60 Fed. Reg. 56,439, at 56,442. 
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WARNING: Where the items are those which can be submitted to complete the entry of the international application into 

the national phase are subsequent to 30 months from the priority date the application is still considered to 
be in the international state and if mailing procedures are utilized to obtain a date the express mail 
procedure of 37 CF.R. §1.10 must be used (since international application papers are not covered by an 
ordinary certificate of mailing - See 37 CF.R. §1.8. 

NOTE: Documents and fees must be clearly identified as a submission to enter the national state under 35 USC 3 71 otherwise 
the submission will be considered as being made under 35 USC 111. 37 CF.R. § 1 494(f). 



1. Applicant herewith submits to the United States Elected Office (EO/US) the following items 
under 35 U.S.C. 371: 

a. [X] This express request to immediately begin national examination procedures (3 5 

U.S.C. 371(f)). 

b. [X] The U.S. National Fee (35 U.S.C. 371(c)(1)) and other fees (37 CF.R. § 1 .492) 

as indicated below: 
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2. Fees 



CT ATMS 
FEE 


(l)FOR 


(2) NUMBER 
FILED 


(3) NUMBER 
EXTRA 


(4) RATE 


(5) CALCULA- 
TIONS 


[ ]* 


TOTAL 
CLAIMS 


28 - 20 = 




x $ 18.00 = 


$ 144.00 




INDEPENDENT 
CLAIMS 


2 -3 = 




x $ 80.00 = 






MULTIPLE DEPENDENT CLAIM(S) (if applicable) + $270.00 




BASIC FEE** 


[ ] U.S. PTO WAS INTERNATIONAL PRELIMINARY EXAMINATION 
AUTHORITY 

Where an International preliminary examination fee as set forth in § 
1 .482 has been paid on the international application to the U.S. PTO: 
[ ] and the international preliminary examination report states that 
the criteria of novelty, inventive step (non-obviousness) and 
industrial activity, as defined in PCT Article 33(2) to (4) have 
been satisfied for all the claims presented in the application 

f>ntprina thf* national ^tapp H7 CFR 1 492faY4Yl $100 00 
entering Hie Ild.LIOIld.1 aldgc \J l v^i rv i .^yt*\a.)\^t }) <4> i 

[ ] and the above requirements are not met (37 CFR 1 .492(a)(1)) 
$690.00 






[x] U.S. PTO WAS NOT INTERNATIONAL PRELIMINARY 
EXAMINATION AUTHORITY 

Where no international preliminary examination fee as set forth 
in § 1.482 has been paid to the U.S. PTO, and payment of an 
international search fee as set forth in § 1.445(a)(2) to the U.S. 
PTO: 

[ ] has been paid (37 CFR 1.492(a)(2)) $710.00 

r I w not been naid C37 CFR 1 492(aY3^ $1,000.00 

[x] where a search report on the international application has been 
prepared by the European Patent Office or the Japanese Patent 
Office (37 CFR 1.492(a)(5)) $860.00 






Total of above Calculations 


=1004.00 


SMALL 
ENTITY 


Reduction by Vz for filing by small entity, if applicable. Statement may also be 
filed, (note 37 CFR 1.9, 1.27, 1.28) 


- 




Subtotal 


1004.00 




Total National Fee 


$1004.00 




Fee for recording the enclosed assignment document $40.00 (37 CFR 1.21(h)). 
(See Item 13 below). See attached "ASSIGNMENT COVER SHEET". 




TOTAL 


Total Fees enclosed 


$1004.00 



*See attached Preliminary Amendment Reducing the Number of Claims. 
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1. 
ii. 



* WARNING: 



w 
[ ] 

A duplicate copy of this sheet is enclosed. 



A check in the amount of 1004.00 
Please charge Account No. 



. to cover the above fees is enclosed, 
in the amount of $ . 



"To avoid abandonment of the application the applicant shall furnish to the United States Patent and 
Trademark Office not later than the expiration of 30 months from the priority date: * * * (2) the basic 
national fee (see § 1.492(a)). The 30-month time limit may not be extended. "37 C.F.R. § 1.495(b). 

WARNING- If the translation of the international application and/or the oath or declaration have not been 

submitted by the applicant within thirty (30) months from the priority date, such requirements may be 
met within a time period set by the Office. 37 C.F.R. § 1.495(b)(2). The payment of the surcharge set 
forth in § 1.492(e) is required as a condition for accepting the oath or declaration later than thirty (30) 
months after the priority date. The payment of the processing fee set forth in § 1.492(f) is required for 
acceptance of an English translation later than thirty (30) months after the priority date. Failure to 
comply with these requirements will result in abandonment of the application. The provisions of § 1.136 
apply to the period which is set. Notice of Jan. 3, 1993, 1147 O.G. 29 to 40. 

3. [x] A copy of the International application as filed (35 U.S.C. 371(c)(2)): 

NOTE. Section 1.495 (b) was amended to require that the basic national fee and a copy of the international application 
must be filed with the Office by 30 months from the priority date to avoid abandonment "The International 
Bureau normally provides the copy of the international application to the Office in accordance with PCT Article 
20. At the same time, the International Bureau notifies applicant of the communication to the Office. In 
accordance with PCT Rule 47.1, that notice shall be accepted by all designated offices as conclusive evidence that 
the communication has duly taken place. Thus, if the applicant desires to enter the national stage, the applicant 
normally need only check to be sure the notice from the International Bureau has been received and then pay the 
basic national fee by 30 months from the priority date. '* Notice of Jan. 7, 1993, 1147 O.G. 29 to 40, at 35-36. See 
item 14c below. 



a. [ ] is transmitted herewith. 

b. [ ] is not required, as the application was filed with the United States Receiving 

Office. 

c. [x] has been transmitted 

i. [x] by the International Bureau. 

Date of mailing of the application (from form PCTYIB/308): . 

ii. [ ] by applicant on . 

Date 



4. 



w 

a. 
b. 
c. 

d. 



A translation of the International application into the English language (35 U.S.C. 
371(c)(2)): 

[ ] is transmitted herewith. 

[x] is not required as the application was filed in English. 

[ ] was previously transmitted by applicant on . 

[ ] will follow. 



Date 
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5. [x] Amendments to the claims of the International application under PCT Article 19 (35 

U.S.C. 371(c)(3)): 

NOTE: The Notice of January 7, 1993 points out that 37 C F.R. § 1.495(a) was amended to clarify the existing and 

continuing practice that PCT Article 19 amendments must be submitted by 30 months from the priority date and 
this deadline may not be extended. The Notice further advises that: "The failure to do so will not result in loss of 
the subject matter of the PCT Article 19 amendments. Applicant may submit that subject matter in a preliminary 
amendment filed under section 1.121. In many cases, filing an amendment under section 1.121 is preferable since 
grammatical or idiomatic errors may be corrected. "1147 O.G. 29-40, at 36. 

a. [ ] are transmitted herewith. 

b. [ ] have been transmitted 

i. [ ] by the International Bureau. 

Date of mailing of the amendment (from form PCT7IB/308): . 

ii. [ ] by applicant on . 

Date 

c. [x] have not been transmitted as 

i. [x] applicant chose not to make amendments under PCT Article 1 9. 
Date of mailing of Search Report (from form PCT/ISA/210): 

11 SEPTEMBER 2000 . 

ii. [ ] the time limit for the submission of amendments has not yet expired. 

The amendments or a statement that amendments have not been 
made will be transmitted before the expiration of the time limit under 
PCT Rule 46.1. 

6. [x] A translation of the amendments to the claims under PCT Article 19 (38 U.S.C. 

371(c)(3)): 

a. [ ] is transmitted herewith. 

b. [ ] is not required as the amendments were made in the English language. 

c. [x] has not been transmitted for reasons indicated at point 5(c) above. 

7. [x] A copy of the international examination report (PCT/IPEA/409) 

[x] is transmitted herewith. 

[ ] is not required as the application was filed with the United States Receiving 
Office. 

8. [x] Annex(es) to the international preliminary examination report 

a. [x] is/are transmitted herewith. 

b. [ ] is/are not required as the application was filed with the United States 

Receiving Office. 

9. [x] A translation of the annexes to the international preliminary examination report 
a. [] is transmitted herewith. 

b- i s not required as the annexes are in the English language. 
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10. [x] An oath or declaration of the inventor (35 U.S.C. 371(c)(4)) complying with 35 

U.S.C. 115 

a. [ ] was previously submitted by applicant on . 

Date 

b. [ ] is submitted herewith, and such oath or declaration 

i. [ ] is attached to the application. 

ii. [ ] identifies the application and any amendments under PCT Article 19 

that were transmitted as stated in points 3(b) or 3(c) and 5(b); and 
states that they were reviewed by the inventor as required by 37 
C.F.R. 1.70. 

c. [x] will follow. 

Other document(s) or information included: 

11. [x] An International Search Report (PCT/IS A/2 1 0) or Declaration under PCT Article 

17(2)(a): 

a. [x] is transmitted herewith. 

b. [ ] has been transmitted by the International Bureau. 

Date of mailing (from form PCT/IB/308): • 

c. [ ] is not required, as the application was searched by the United States 

International Searching Authority. 

d. [ ] will be transmitted promptly upon request. 

e. [ ] has been submitted by applicant on . 

Date 

12. [x] An Information Disclosure Statement under 37 C.F.R. 1 .97 and 1 .98: 

a. [ ] is transmitted herewith. 

Also transmitted herewith is/are: 
[ ] Form PTO-1449 (PTO/SB/08A and 08B). 
[ ] Copies of citations listed. 

b. [x] will be transmitted within THREE MONTHS of the date of submission of 

requirements under 35 U.S.C. 371(c). 

c. [ ] was previously submitted by applicant on . 

Date 

13. [ ] An assignment document is transmitted herewith for recording. 

A separate [ ] "COVER SHEET FOR ASSIGNMENT (DOCUMENT) ACCOMPANYING 
NEW PATENT APPLICATION" or [ ] FORM PTO 1595 is also attached. 
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Additional documents: 


a. 


[ ] 


Copy of request (PCT/RO/101) 


b. 


[x] 


International Publication No. WO 00/45599 




i. 


[x] Specification, claims and drawing 




ii. 


[ ] Front page only 


c. 


[ 1 


Preliminary amendment (37 C.F.R. § 1.121) 


d. 


W 


Other 



FORM PCT/IPEA/409; RESPONSE TO WRITTEN OPINION 



15. [x] The above checked items are being transmitted 

a. [x] before 30 months from any claimed priority date. 

b. [ ] after 30 months. 

16. [ ] Certain requirements under 35 U.S.C. 371 were previously submitted by the 

applicant on 

, namely: 



AUTHORIZATION TO CHARGE ADDITIONAL FEES 

Accurately count claims, especially multiple dependent claims, to avoid unexpected high charges if 
extra claims are authorized. 

NOTE: "A written request may be submitted in an application that is an authorization to treat any concurrent or future 
reply, requiring a petition for an extension of time under this paragraph for its timely submission, as 
incorporating a petition for extension of time for the appropriate length of time. An authorization to charge all 
required fees, fees under §1.17, or all required extension of time fees will be treated as a constructive petition for 
an extension of time in any concurrent or future reply requiring a petition for an extension of time under this 
paragraph for its timely submission. Submission of the fee set forth in § 1.17(a) will also be treated as a 
constructive petition for an extension of time in any concurrent reply requiring a petition for an extension of time 
under this paragraph for its timely submission. "37 C.F.R. § 1.136(a)(3). 

NOTE: "Amounts of twenty-five dollars or less will not be returned unless specifically requested within a reasonable 

time, nor will the payer be notified of such amounts; amounts over twenty-five dollars may be returned by check 
or, if requested, by credit to a deposit account. "37 C.F.R. § 1.26(a). 

[X] The Commissioner is hereby authorized to charge the following additional fees that 
may be required by this paper and during the entire pendency of this application to 
Account No. 12-0425 



WARNING: 



[X] 37 C.F.R. 1.492(a)(1), (2), (3), and (4) (filing fees) 

WARNING: Because failure to pay the national fee within 30 months without extension (37 C.F.R. § 1.495(b)(2)) 

results in abandonment of the application, it would be best to always check the above box. 

[ ] 37 C.F.R. 1.492(b), (c) and (d) (presentation of extra claims) 

NOTE: Because additional fees for excess or multiple dependent claims not paid on filing or on later presentation must 
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only be paid or these claims cancelled by amendment prior to the expiration of the time period set for response by 
the PTO in any notice of fee deficiency (37 C.F R. § 1. 4 92(d)), it might be best not to authorize the PTO to charge 
additional claim fees, except possible when dealing with amendments after final action. 



NOTE: Where an authorization to charge the issue fee to a deposit account has been filed before the mailing of a Notice 



of Allowance, the issue fee will be automatically charged to the deposit account at the time of mailing the notice 



of allowance. 37 C.F.R. § 1311(b). 

NOTE: 37 C.F.R. 1.28(b) requires 'Notification of any change in loss of entitlement to small entity status must be filed in 
the application . . . prior to paying, or at the time of paying . . . issue fee. " From the wording of 37 C.F.R. § 
1.28(b): (a) notification of change of status must be made even if the fee is paid as "other than a small entity" and 
(b) no notification is required if the change is to another small entity. 



[X] 
[X] 
[X] 



37 C.F.R. 1.17 (application processing fees) 

37 C.F.R. 1.17(a)(l)-(5)(extension fees pursuant to § 1.136(a). 

37 C.F.R. 1.18 (issue fee at or before mailing of Notice of Allowance, 

pursuant to 37 C.F.R. 1.31 1(b)) 



[ ] 



37 C.F.R. § 1.492(e) and (f) (surcharge fees for filing the declaration and/or 
filing an English translation of an InternationaLApplication later than 30 
months after the priority date). sfjr 




URE OF PRACTITIONER 



Reg. No.: 25,858 



WILLIAM R. EVANS 



(type or print name of practitioner) 



Tel. No.: (212) 708-1930 



P.O. Address 



Customer No.: 00140 



c/o Ladas & Parry 
26 West 61 st Street 
New York, N.Y. 10023 



(Transmittal Letter to the United States Elected Office (EOAJS) — page 8 of 8) 13-18 



Rec'd PCT/PT8 0 8 NOV 200! 



PATENT 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re application of: Patrick J. RAINSFORD 
Serial No.: 09/889,966 Group No.: - 

Filed: July 24, 2001 Examiner.: - 

For: INTERACTIVE SYSTEM 

Attorney Docket No.: U 013563-5 

Assistant Commissioner for Patents 
Washington, D.C. 20231 

Sir: 

PRELIMINARY AMENDMENT 
Please amend the application as follows: 
IN THE CLAIMS 

3. (Amended) An apparatus as claimed in claim 1, wherein said means for parsing 
include means for inputting criteria to be used to recognise a change of shot. 

4. (Amended) An apparatus as claimed in claim 1, wherein the means for extracting 
attributes of the object includes means for isolating the object within a boundary formed on 

CERTIFICATE OF MAILING (37 CFR 1.8a) 

I hereby certify that this paper (along with any paper referred to as being attached or enclosed) 
is being deposited with the United States Postal on the date shown below with sufficient postage 
as first class mail in an envelope addressed to the: Assistant Commissioner for Patents, 
Washington, D.C. 20231 

CLIFFORD J. MASS 
Type or print name of person mailing paper) 

Date: Aueust 30. 2001 



(Signature of person mailing paper) 



the frame, means for performing edge detection within the boundary to identify and locate edges 
of said object, and storing means for storing a geometric model of said object. 

5. (Amended) An apparatus as claimed in claim 1, wherein said means for extracting 
attributes of said object also includes means for recording at least one of the attributes of shape, 
size, position, colour, texture, intensity gradient of said object, and time series statistics based 
on said attributes. 

6. (Amended) An apparatus as claimed in claim 1, wherein said means for extracting 
attributes of said object includes means for comparing said attributes of said object with 
attributes of objects previously stored to determine whether the object is distinguishable 
therefrom, and when said object is determined not to be distinguishable, providing means for 
re-defining the object. 

1. (Amended) An apparatus as claimed in claim 1, wherein said means for extracting 
attributes of said object includes means for comparing the location in the frame of said object 
with the location of objects already stored for that frame to determine whether that object is 
distinguishable therefrom, and where the location of said object is not distinguishable from the 
location of another object providing means for assigning rank to the objects to determine which 
object will be associated with that location. 

8. (Amended) An apparatus, as claimed in claim 1, wherein the means for utilising the 
attributes of the object for tracking the object includes means for updating the stored attributes 
of the object as the attributes of the object change from frame to frame. 
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9. (Amended) An apparatus as claimed in claim 1 , wherein said tracking means utilising 
the attributes of the object for tracking the object includes plural algorithm means for calculation 
of independent tracks of objects for use depending on the visual 

complexity of a sequence to automatically track said objects in different types of visual 
environment. 

10. (Amended) An apparatus as claimed in claim 1, wherein said tracking means for 
utilising the attributes of the object for tracking the object includes means for converting all the 
frames to be tracked to a low-level representation, means for determining the position of each 
object in the frames by minimising a distance measure to locate each object in each frame, means 
for processing the positions of said object to smooth over occlusions and the entrances and exits 
of objects into and out of said frames, and means for reviewing the object within a tracked 
sequence and for correcting the location attributes of any misplaced objects. 

1 1 . (Amended) An apparatus, as claimed in claim 1 , wherein the means for associating 
interactive data with the object in the key-frame includes means for providing a database of 
different types of data including one or more of URLs, HTML pages, video clips, audio clips, 
text files and multimedia catalogues, and means for selecting said interactive content data from 
the database to associate with said object. 

12. (Amended) An apparatus, as claimed in claim 1, wherein the means for associating 
interactive data with the object in the key- frame produces said data sequence using means for 
determining whether the embedded interactive content data is frame synchronous 

data associated with object positions, shapes, ranks and pointers in a frame, or 
group-synchronous data associated with all the objects in a group, or is data to be streamed just 
in time, wherein means are provided for associating frame synchronous data with the 
corresponding frame, means are provided for associating group synchronous data with the frame 
at which a group changes, and means are provided for streaming just in time data to a user before 
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it is required to be associated with the corresponding objects. 

13. (Amended) An apparatus as claimed in claim 1, wherein means are provided to 
associate different interactive content data with respectively different objects. 

16. (Amended) A method as claimed in claim 14, wherein step c) includes the step of 
producing a hierarchy of groups of sequences of shots. 

17. (Amended) A method as claimed in claim 14, wherein step e) includes the steps of: 
isolating the object within a boundary formed on the frame, performing edge detection within the 
boundary to identify and locate edges of said object, and step f) includes storing a geometric 
model of said object. 

18. (Amended) A method as claimed in claim 14, wherein step f) includes the step of 
recording at least one of the attributes of shape, size, position, colour, texture, intensity gradient 
of said object, and time series statistics based on said attributes. 

19. (Amended) A method as claimed in claim 14, wherein step f) includes the step 
of comparing said attributes of said object with attributes of objects previously stored to 
determine whether the object is distinguishable therefrom, and when said object is determined 
not to be distinguishable, the step of redefining the object. 

20. (Amended) A method as claimed in claim 14, wherein step f) includes the step of 
comparing the location in the frame of said object with the location of objects already stored for 
that frame to determine whether that object is distinguishable therefrom, and where the location 
of said object is not distinguishable from the location of another object, the step of aligning rank * 
to the objects to determine which object will be associatecN*/ith that location. 
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21 . (Amended) A method, as claimed in claim 14, wherein step h) includes the step of 
updating the stored attributes of the object as the attributes of the object change from frame to 
frame. 

22. (Amended) A method as claimed in claims 14, wherein step h) includes the step of 
using plural algorithm means for calculation of independent tracks of objects for use depending 
on the visual complexity of a sequence automatically to track said objects in different types of 
visual environment. 

23. (Amended) A method as claimed in claim 14, wherein step h) includes the steps of 
converting all the frames to be tracked to a low- level representation, determining the position of 
each object in the frames by minimising a distance measure to locate each object in each frame, 
processing the positions of said object to smooth over occlusions and the entrances and exits of 
objects into and out of said frames, reviewing the object within a tracked sequence and correcting 
the location attributes of any misplaced objects. 

24. (Amended) A method, as claimed in claim 14, wherein step g) includes the steps of 
providing a database of different types of data including one or more of URLs, HTML pages, 
video clips, audio clips, text files and multimedia catalogues, and selecting said interactive 
content data from the database to associate with said object. 

25. (Amended) A method, as claimed in claim 14, wherein step j) includes determining 
whether the embedded interactive content data is frame synchronous data associated with object 
positions, shapes, ranks and pointers in a frame, or group-synchronous data associated with all 
the objects in a group, or is data to be streamed just in time, and associating frame synchronous 
data with the corresponding frame, associating group synchronous data with the frame at which 
a group changes, and streaming just in time data to a user before it is required to be associated 
with the corresponding objects, 



respectively. 



26. (Amended) A method as claimed in claim 14, wherein in steps d) to j) different 
interactive content data are associated with respectively different objects. 

27. (Amended) A computer program comprising code means for performing all the steps 
of the method of claim 14 when the program is run on one or more computers. 

REMARKS 

The above amendatory action is taken solely for the purpose of avoiding claim fees that 
would otherwise accrue due to the presence of multiple dependent claims. 

Respectfully submitted, 



Clifford J. Mass 

c/o Ladas & Parry 

26 West 61 st Street 

New York, New York 10023 

Reg. No. 30,086 Tel. No. (212) 708-1890 
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3. (Amended) An apparatus as claimed in [claims] claim 1 [or 2], wherein said means for 
parsing include means for inputting criteria to be used to recognise a change of shot. 

4. (Amended) An apparatus as claimed in [any of claims] claim 1, [to 3] wherein the means 
for extracting attributes of the object includes means for isolating the object within a boundary 
formed on the frame, means for performing edge detection within the boundary to identify and locate 
edges of said object, and storing means for storing a geometric model of said object. 

5 . (Amended) An apparatus as claimed in [any of claims] claim 1 , [to 4] wherein said means 
for extracting attributes of said object also includes means for recording at least one of the attributes 
of shape, size, position, colour, texture, intensity gradient of said object, and time series statistics 
based on said attributes. 

6. (Amended) An apparatus as claimed in [any of the preceding claims] claim 1, wherein 
said means for extracting attributes of said object includes means for comparing said attributes of 
said object with attributes of objects previously stored to determine whether the object is 
distinguishable therefrom, and when said object is determined not to be distinguishable, providing 
means for re-defining the object. 

7. (Amended) An apparatus as claimed in [any of the preceding claims] claim 1 , wherein 
said means for extracting attributes of said object includes means for comparing the location in the 
frame of said object with the location of objects already stored for that frame to determine whether 
that object is distinguishable therefrom, and where the location of said object is not distinguishable 
from the location of another object providing means for assigning rank to the objects to determine 
which object will be associated with that location. 



8, (Amended) An apparatus, as claimed in [any of the preceding claims] claim 1 . wherein 
the means for utilising the attributes of the object for tracking the object includes means for updating 
the stored attributes of the object as the attributes of the object change from frame to frame. 



9. (Amended) An apparatus as claimed in [any of the preceding claims] claim 1 , wherein 
said tracking means utilising the attributes of the object for tracking the object includes plural 
algorithm means for calculation of independent tracks of objects for use depending on the visual 
complexity of a sequence to automatically track said obj ects in different types of visual environment. 

10. (Amended) An apparatus as claimed in [any of the preceding claims] claim L wherein 
said tracking means for utilising the attributes of the object for tracking the object includes means 
for converting all the frames to be tracked to a low-level representation, means for determining the 
position of each object in the frames by minimising a distance measure to locate each object in each 
frame, means for processing the positions of said object to smooth over occlusions and the entrances 
and exits of objects into and out of said frames, and means for reviewing the object within a tracked 
sequence and for correcting the location attributes of any misplaced objects. 

1 1 . ( Amended) An apparatus, as claimed in [any of the preceding claims] claim 1 , wherein 
the means for associating interactive data with the object in the key- frame includes means for 
providing a database of different types of data including one or more of URLs, HTML pages, video 
clips, audio clips, text files and multimedia catalogues, and means for selecting said interactive 
content data from the database to associate with said object. 

12. (Amended) An apparatus, as claimed in [any of the preceding claims] claim 1 % wherein 
the means for associating interactive data with the object in the key-frame produces said data 
sequence using means for determining whether the embedded interactive content data is frame 
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synchronous data associated with object positions, shapes, ranks and pointers in a frame, or 
group-synchronous data associated with all the objects in a group, or is data to be streamed just in 
time, wherein means are provided for associating frame synchronous data with the corresponding 
frame, means are provided for associating group synchronous data with the frame at which a group 
changes, and means are provided for streaming just in time data to a user before it is required to be 
associated with the corresponding objects. 

13. (Amended) An apparatus as claimed in [any of the preceding claims] claim L wherein 
means are provided to associate different interactive content data with respectively different objects. 

16. (Amended) A method as claimed in [claims] claim 14, [to 16] wherein step c) includes 
the step of producing a hierarchy of groups of sequences of shots. 

17. (Amended) A method as claimed in [any of claims] claim 14, [to 16] wherein step e) 
includes the steps of: isolating the object within a boundary formed on the frame, performing edge 
detection within the boundary to identify and locate edges of said object, and step f) includes storing 
a geometric model of said object. 

18. (Amended) A method as claimed in [any of claims] claim 14, [to 17] wherein step f) 
includes the step of recording at least one of the attributes of shape, size, position, colour, texture, 
intensity gradient of said object, and time series statistics based on said attributes. 

19. (Amended) A method as claimed in [any of claims] claim 14, [to 18] wherein step f) 
includes the step of comparing said attributes of said object with attributes of objects previously 
stored to determine whether the object is distinguishable therefrom, and when said object is 
determined not to be distinguishable, the step of redefining the object. 
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20. (Amended) A method as claimed in [any of claims] claim 14, [to 19] wherein step f) 
includes the step of comparing the location in the frame of said object with the location of objects 
already stored for that frame to determine whether that object is distinguishable therefrom, and where 
the location of said object is not distinguishable from the location of another object, the step of 
assigning rank to the objects to determine which object will be associated with that location. 

21 . (Amended) A method, as claimed in [any of claims] claim 14, [to 20] wherein step h) 
includes the step of updating the stored attributes of the object as the attributes of the object change 
from frame to frame. 

22. (Amended) A method as claimed in [any of claims] claim 14, [to 21] wherein step h) 
includes the step of using plural algorithm means for calculation of independent tracks of objects for 
use depending on the visual complexity of a sequence automatically to track said objects in different 
types of visual environment. 

23. (Amended) A method as claimed in [any of claims] claim 14, [to 22] wherein step h) 
includes the steps of converting all the frames to be tracked to a low-level representation, 
determining the position of each object in the frames by minimising a distance measure to locate 
each object in each frame, processing the positions of said object to smooth over occlusions and the 
entrances and exits of objects into and out of said frames, reviewing the object within a tracked 
sequence and correcting the location attributes of any misplaced objects. 

24. (Amended) A method, as claimed in [any of claims] claim 14, [to 23] wherein step g) 
includes the steps of providing a database of different types of data including one or more of URLs, 
HTML pages, video clips, audio clips, text files and multimedia catalogues, and selecting said 
interactive content data from the database to associate with said object. 
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25. (Amended) A method, as claimed in [any of claims] claim 14, [to 24] wherein step j) 
includes determining whether the embedded interactive content data is frame synchronous data 
associated with object positions, shapes, ranks and pointers in a frame, or group-synchronous data 
associated with all the objects in a group, or is data to be streamed just in time, and associating frame 
synchronous data with the corresponding frame, associating group 

synchronous data with the frame at which a group changes, and streaming just in time data to a user 

before it is required to be associated with the corresponding objects, 

respectively. 

26. (Amended) A method as claimed in [any of claims] claim 14, [to 25] wherein in steps 
d) to j) different interactive content data are associated with respectively different objects. 

27. (Amended) A computer program comprising code means for performing all the steps 
of the method of [any of claims] claim 14 [to 26] when the program is run on one or more 
computers. 
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I. IDENTIFICATION AND RIGHTS AS A SMALL ENTITY 

I hereby state that I am 



(complete either (a), (b), (c) or (d) below) 



(a) 



Independent Inventor 

[ ] a below named independent inventor, and that I qualify as an independent 
inventor, as defined in 3 7 CFR 1 .9(c), for purposes of paying reduced fees under 
Sections 41(a) and (b) of Title 35, United States Code, to the Patent and 
Trademark Office, 
(b) Noninventor Supporting a Claim by Another 

[ ] making this statement to support a claim by 



for a small entity status for purposes of paying reduced fees under Sections 41(a) and (b) of Title 35, 
United States Code. I hereby state that I would qualify as an independent inventor as defined in 37 CFR 
1 .9(c) for purposes of paying reduced fees under Sections 41(a) and (b) of Title 35, United States Code, 
if I had made the above identified invention. 

Small Business Concern 

[ ] the owner of the small business concern identified below: 

[ ] an official of the small business concern empowered to act on behalf of the concern 
identified below: 



(c) 

check 
one -+ 
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Name of Concern EMUSE CORPORATION 

Address of Concern 80 Park Avenue. Sandvmount Dublin 4. IRELAND 

and 

that the above identified small business concern qualifies as a small business concern, as defined in 13 
CFR 121.3-18, and reproduced in 37 CFR 1.9(d), for purposes of paying reduced fees under Sections 
41(a) and (b) of Title 35, United States Code, in that the number of employees of the concern, including 
those of its affiliates, does not exceed 500 persons. For purposes of this statement, (1) the number of 
employees of the business concern is the average over the previous fiscal year of the concern of the 
persons employed on a full-time, part-time or temporary basis during each of the pay periods of the fiscal 
year, and (2) concerns are affiliates of each other when either, directly or indirectly, one concern controls 
or has the power to control the other, or a third party or parties controls or has the power to control both. 

(d) Non-Profit Organization 

[ ] an official empowered to act on behalf of the nonprofit organization identified below: 

Name of Organization 

Address of Organization 



TYPE OF ORGANIZATION 

[ ] University or Other Institution of Higher Education 

[ ] Tax Exempt Under Internal Revenue Service Code (26 USC 501(a) and 501(c) (3)) 

[ ] Nonprofit Scientific or Educational Under Statute of State of the United States of 
America 

(Name of State ) 

(Citation of Statute ) 

[ ] Would Qualify as Tax Exempt Under Internal Revenue Service Code (26 USC 501(a) 
and 501(c) (3)), if Located in the United States of America 

[ ] Would Qualify as Nonprofit Scientific or Educational Under Statute of State of the 
United States of America, if Located in the United States of America 

(Name of State ) 

(Citation of Statute ) 

and that the nonprofit organization identified above qualifies as a nonprofit organization, as defined in 
37 CFR 1.9(e), for purposes of paying reduced fees under Sections 41(a) and (b) of Title 35, United 
States Code. 

n. OWNERSHIP OF INVENTION BY DECLARANT 

I hereby state that rights under contract or law remain with and/or have been conveyed to the 
above identified 

[ ] person [x] concern [ ] organization 

(item (a) or (b) above) (item (c) above) (item (d) above) 
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EXCEPT, that if the rights held are not exclusive, each individual, concern or organization having rights 
to the invention is listed below* and no rights to the invention are held (1) by any person who could not 
be classified as an independent inventor under 37 CFR 1 .9(c) if that person had made the invention, (2) 
any concern which would not qualify as a small business concern under 37 CFR 1 .9(d) or (3) a nonprofit 
organization under 37 CFR 1.9(e). 

[x] no such person, concern, or organization 

[ ] person, concerns or organizations listed below* 

*NOTE: Separate statements are required from each named person, concern or organization having rights to the invention 
as to their status as small entities. (37 CFR 1.27) 



Full Name 

Address 

[ ] INDIVIDUAL [ ] SMALL BUSINESS CONCERN [ ] NONPROFIT ORGANIZATION 

Full Name 

Address 

[ ] INDIVIDUAL [ ] SMALL BUSINESS CONCERN [ ] NONPROFIT ORGANIZATION 



m. ACKNOWLEDGEMENT OF DUTY TO NOTIFY PTO OF STATUS CHANGE 

I acknowledge the duty to file, in this application or patent, notification of any change in status 
resulting in loss of entitlement to small entity status prior to paying, or at the time of paying, the earliest 
of the issue fee or any maintenance fee due after the date on which status as a small entity is no longer 
appropriate. (37 CFR 1.28(b)) 

IV. DECLARATION 

(check the following item, if desired) 

NOTE: The following verification statement need not be made in accordance with the rules published on October 10, 1997, 
62 Fed. Reg. 52131, effective December 1, 1997. 

NOTE: "The presentation to the Office (whether by signing, filing, submitting, or later advocating) of any paper by a party, 
whether a practitioner or non-practitioner, constitutes a certification under § 10.18(b) of this chapter. Violations of 
§10. 18(b)(2) of this chapter by a party, whether a practitioner or non-practitioner, may result in the imposition of 
sanctions under § 10 J 8(c) of this chapter. Any practitioner violating § 10.18(b) may also be subject to disciplinary 
action. See §§ 10.18(d) and 10.23(c)(15). " 37 CFR 1.4(d)(2). 

[ ] I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 1 8 of the United States 
Code, and that such willful false statements may jeopardize the validity of the application, any 
patent issuing thereon, or any patent to which this verified statement is directed. 
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V. SIGNATURES 

(complete only (e) or (f) below) 

(e) 

NOTE: All inventors must sign the statement. 



Name of Inventor 



Date: 



Signature of Inventor 



Name of Inventor 



Date: 



Signature of Inventor 



Name of Inventor 



Date: 



Signature of Inventor 



(add lines for any additional inventors who must sign) 



or 



(f> 

NOTE: The title of the person signing oryi&half of a conc&f% or nonprofit organization should be specified. 
Name of Person Signin g (x) 

Title of Person (x) * & • ° • 

(if signing on behalf of a concern or non-profit organization) 

Address of Person Signin g Emuse Corporation. 80 Park Avenue, Sandvmount Dublin 4, IRELAND 
SIGNATURE £x) DATE (x) 
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I 

INTERACTIVE SYSTEM 

This invention relates to an interactive system and 
particularly to a system for multiplexing data in a digital video 
5 signal. 

It is known to provide a video programme in the form of a 
digital signal which may be broadcast , or which may be provided 
on a digital video disk (DVD) or a video tape and the present 
invention is not restricted to the form in which the video signal 

10 for a programme is provided. 

With the increasing number of television broadcasting 
channels, there is a dilution of advertising revenue since, for 
commercial reasons, an advertiser restricts their marketing 
effort to a limited number of broadcast channels. In addition, 

15 there is an increase in availability of devices available to a 
viewer for preventing the reception of unwanted advertisements, 
e.g. a V-chip, but at the present time there is currently no way 
of selectively blocking advertisements, with the result that 
those advertisements that may be of interest to a viewer are also 

20 blocked . 

With the growing use of the Internet, users are becoming 
accustomed to having access to large and diverse sources of data 
and information using a personal computer (PC) or, for example, a 
digital set-top box used in conjunction with a television and 

25 remote control or mouse. 

The present invention seeks to provide a system which 
enables a viewer to interact with a video signal which may be 
broadcast so as- to facilitate information transfer and/or 
transactions that may be performed over the Internet* 

30 According to one aspect of this invention there is provided 

an interactive system including means for providing a video 
programme signal, means for generating interactive content data 
associated with at least one object, said data being associated 
with frames of said video programme signal in which the object 

35 appears, means for multiplexing said data with said video 

programme signal, means for viewing the video programme signal, 
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means for retrieving said data and means for using said data to 
obtain details of said object. 

Preferably, said means for using include means for accessing 
an interactive Web site to obtain said details of said object. 
Conveniently, said means for using further include means for 
producing a list of details of said object and means for 
selecting from said list* 

Advantageously, said means for accessing an interactive Web 
site is adapted to secure details of said object which may 
include a purchasing transaction for said object or browsing an 
advertising catalogue. 

Preferably, the means for generating includes means for 
tracking said object in each frame of said video programme signal 
in which said object appears and means for identifying the 
location of said object in each said frame. 

Preferably, each frame of said video programme includes said 
interactive content data. 

Advantageously, said tracking means includes means for 
determining scene breaks and means for searching for said object 
in a next frame in which said object appears. 

Conveniently, said multiplexing means includes means for 
synchronising said data with audio and video data of said 
programme signal to generate a transport stream, for example, a 
MPEG -2 /DVB transport stream. 

Advantageously, said system includes means for broadcasting 
said transport stream via, for example, at least one of a 
satellite, terrestrial and cable network. 

Conveniently, said means for retrieving includes one of a 
mouse, a keyboard, and remote control device. 

According to a second aspect of this invention there is 
provided apparatus for associating data representative of an 
object with a digital video programme including means for 
providing a digital video programme having plural individual 
frames at least some of which incorporate said object, means for 
selecting a frame of the video programme in which said object 
appears to provide a key-frame, means for selecting said object 
within the key-frame with which data is to be associated, means 
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for extracting attributes of the object from the key-frame f means 
for associating interactive data with the object in the key- 
frame, means for utilising the attributes of the object for 
tracking the object through subsequent frames of the video 
programme, whereby said interactive data is associated with the 
object in subsequent frames of the video programme in which said 
object has been tracked and said interactive content data is 
embedded with data representative of said object in a data 
sequence • 

Advantageously, means are provided for converting said data 
sequence to a standard data sequence, for example, an MPEG-2/DVB 
compliant data sequence. 

Where the video programme is in an analogue format means are 
preferably provided for converting the programme to digitised 
form* 

Preferably, the means for selecting a frame of the video 
programme includes means for producing an edit list to divide the 
digitised video programme into a plurality of sequences of 
related shots, 'and means for selecting at least one key-frame 
from within each sequence ♦ 

Advantageously, the means for producing an edit list further 
includes means for parsing the video programme by identifying 
separate shots in the video programme to produce the edit list, 
means for identifying shots containing related content to form a 
sequence of shots containing related content, and means for 
producing a hierarchy of groups of shots. 

Advantageously, said means for parsing include means for 
inputting criteria to be used to recognise a change of shot. 

Preferably, the means for extracting attributes of the 
object includes means for isolating the object within a boundary 
formed on the frame, means for performing edge detection within 
the boundary to identify and locate edges of said object, and 
storing means for storing a geometric model of said object. 

Conveniently, said means for extracting attributes of said 
object also includes means for recording at least one of the 
attributes of shape, size, position, colour, texture, intensity 
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gradient of said object, and time series statistics based on said 
attributes . 

Advantageously, said means for extracting attributes of said 
object includes .means for comparing said attributes of said 
object with attributes of objects previously stored to determine 
whether the object is distinguishable therefrom, and when said 
object is determined not to be distinguishable, providing means 
for re-defining the object, for example by re-defining said 
boundary. 

Preferably r said means for extracting said attributes 
includes means for comparing the location in the frame of said 
object with the location of objects already stored for that frame 
to determine whether that object is distinguishable therefrom, 
and where the location of said object is not distinguishable from 
the location of, another object providing means for assigning rank 
to the objects to determine which object will be associated with 
that location. 

Preferably; the means for tracking the object includes means 
for updating the stored attributes of the object as the object 
moves location within different frames. 

Advantageously, said means for tracking includes plural 
algorithm means for use depending on the visual complexity of a 
sequence to automatically track objects in different types of 
visual environment. 

Advantageously, said tracking means includes means for 
converting all the frames to be tracked to a low-level 
representation, means for determining the position of each object 
in the frames by minimising a distance measure to locate each 
object in each frame, means for processing the positions of said 
object to smooth over occlusions and the entrances and exits of 
objects into and out of said frames, and means for reviewing the 
object within a tracked sequence and for correcting the location 
attributes of any misplaced objects. 

Preferably, the means for associating includes means for 
providing a database of different types of data including one or 
more of URLs , HTML pages, video clips, audio clips, text files 
and multimedia catalogues, and means for selecting said 
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interactive content data from the database to associate with said 
object. 

Preferably, the means for associating produces said data 
sequence using means for determining whether the embedded 
interactive content data is frame synchronous data associated 
with object positions, shapes, ranks and pointers in a frame, or 
group- synchronous data associated with all the objects in a 
group, or is data to be streamed just in time, wherein means are 
provided for associating frame synchronous data with the 
corresponding frame, means are provided for associating group 
synchronous data with the frame at which a group changes, and 
means are provided for streaming just in time data to a user 
before it is required to be associated with the corresponding 
ob j ects . 

It will be understood that although the above has been 
defined in relation to associating interactive content data with 
one object, different interactive content data may be associated 
with respectively different objects. 

According to a third aspect of this invention there is 
provided apparatus for embedding a data sequence within a generic 
digital transport stream (such as DVB/MPEG-2 or ATSC/MPEG-2) 
including means for receiving a data sequence of interactive 
content data associated with an object in a digitised video 
signal, means for synchronising the data sequence with the video 
and audio of the digitised video signal to generate a further 
transport stream, and means for associating a packet identifier 
with the further transport stream. 

In a preferred embodiment, means are provided for 
broadcasting the further transport stream to viewers. 

Preferably, the means for receiving a data sequence includes 
means for receiving elementary streams comprising a digital video 
signal stream, a digital audio stream, a digital data sequence 
stream and a digital control data stream, means for packetising 
each of the data streams into fixed size blocks and adding a 
protocol header to produce packet ised elementary streams, and 
means for synchronising the packetised elementary streams with 
time stamps to establish a relationship between the data streams. 
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Preferably, the means for synchronising the data sequence 
includes means for multiplexing packetised elementary streams 
into transport packets headed by a synchronisation byte, and 
means for assigning a different packet identifier to each 
packetised elementary stream. 

Advantageously, means for synchronising the packetised 
elementary streams with time stamps includes means for stamping 
with a reference time stamp to indicate current time, and means 
for stamping with a decoding time stamp to indicate when the data 
sequence stream has to be synchronised with the video and audio 
streams ♦ 

Conveniently, the means for broadcasting the further 
transport stream to users includes means for providing a 
programme association table listing all the channels to foe 
available in the broadcast, means for providing a programme map 
table identifying all the elementary streams in the broadcast 
channel, and means for transmitting the programme association 
table and the programme map table as separate packets within the 
further transport stream. 

According 4 to a fourth aspect of this invention there is 
provided apparatus for retrieving data embedded in a generic 
digital transport stream in which the embedded data includes a 
data sequence of data associated with objects represented by the 
generic digital transport stream, said apparatus including means 
for recognising a packet identifier within the video signal, 
means for extracting the data sequence from the generic digital 
transport stream, means for identifying objects within the video 
sequence from which to retrieve associated data, means for 
synchronising said data sequence to said identified objects and 
means for interactively using said associated data. 

Preferably, said means for identifying objects includes 
means for selecting an object within a frame, means for 
displaying data associated with said object, means for selecting 
data from a list of displayed data, and means for extracting the 
embedded data associated with the data relating to said object. 

Conveniently, means are provided for selecting a frame to 
display the objects having embedded associated data, means for 
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selecting one of the displayed objects to display a list of the 
data associated with said object, and means for selecting from 
said list. 

Conveniently, the means for selecting includes means for 
storing the frame for subsequent display and subsequent recall of 
the frame . 

In a preferred embodiment, the extracted embedded data is 
applied to means for accessing an Internet web site to facilitate 
interactive communication such as e-commerce. 

By using the present invention, advertisements produced by 
advertisers are unobtrusive, i.e. the viewer can watch the 
programme without interacting, if so desired. Alternatively, the 
viewer can view the programme and freeze a frame of the 
programme, click on an object using a mouse, keyboard or TV 
remote control and, over the Internet, facilitate an e-commerce 
transaction. In performing such a function the viewer may split 
the VDU screen so that one portion continues to display the 
running programme and another portion displays the frozen frame 
and the Internet information transfer. 

The invention can be used in numerous aspects of digital 
video entertainment, especially broadcasting, i.e. 
1. Interactive product placement in regular television 

programmes or movies. 
2* Fashion TV* 
3* Music TV. 

4* Educational programmes. 

The e-commerce may facilitate, for example, merchandising to 
ticket sales. 

The invention has the advantage that a viewer is able to 
select further' information on those items of interest within a 
video signal programme without being overwhelmed with information 
of no relevance. This is particularly useful where the 
information is in the form of advertisements and is achieved by 
making objects viewed in the video programme have associated 
multiplexed (embedded) data to provide links to further 
information relevant to those objects, either to information 
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within the video signal or stored in a database or by accessing 
an Internet web site. 

As far as the advertiser is concerned, the invention has the 
advantage that advertisements can be precisely targeted to a 
relevant audience and the advertisements cannot be stopped from 
reaching the user by a device for blocking out advertisements, 
e.g. a V-chip. Because multiple advertisers may associate their 
advertisements with each frame of a video programme sequence, the 
invention has the potential of reducing the costs of advertising 
to individual advertisers while maintaining or increasing 
advertising revenues for programme makers and suppliers. In this 
way, data-carrying potential of each frame of a video programme 
signal may be maximised and maximum use of the data-carrying 
capacity of broadcast channels may be achieved. The present 
invention is believed to lead the way to generating a new 
democracy for advertisers that may not be able to afford, for 
example, a two minute segment on broadcast TV at peak times. 
This is because the present invention allows multiple advertisers 
per object, and/or multiple objects per frame, leading to a high 
level of flexibility in advertising revenue models. 

In the field of, for example, music videos, the content may 
be used to promote the music of the band for the record label and 
by interacting with the musicians, a user may purchase and 
download the music directly. 

Additionally, plural advertisers may be buying the same slot 
- in other words, the advertiser's content is totally fused 
Within the programme content and it is not until the advertising 
content is downloaded by the user that it is read. Thus, every 
frame of a digital TV programme may be used as advertising 
revenue* An e-commerce database may store all relevant data 
concerning the advertisers, from URL addresses of Web sites to 
catalogues, brochures and video promotions, to e-commerce 
transaction facilities. 

When a viewer selects an object by, for example, using a 
mouse to click on the object, that object may represent a number 
of advertisers, e.g. a musician may advertise clothing, a watch, 
cosmetics, and a musical instrument, so that the viewer selects 
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from a list of promoted items associated with the object. There 
is, thus, presented a push technology approach which maximises 
the transmission, speed of a satellite broadcast. The user needs 
only a return path via the Internet if he actually wishes to 
carry out a transaction. 

The invention will now be described, by way of example, with 
reference to the accompanying drawings, in which: 

Figure 1 shows a block schematic diagram of an interactive 
system of this invention, 

Figure 2 shows a block schematic diagram of video programme 
processing for generating interactive content data associated 
with an object in relevant frames of a programme, 

Figure 3 shows a schematic diagram indicating programme 
sequences derived by groups of related camera shots, 

Figure 4 shows a block schematic diagram of a parser shown 
in Figure 2, whereby groups of shots are produced, 

Figure 5 shows a key frame of a video programme, 

Figure 6 shows an object selected in the key frame of Figure 

5, 

Figure 7 shows a flow diagram for frame by frame 
identification of objects in a video programme, 

Figure 8 shows a flow diagram of the object tracker shown in 
Figure 2 for tracking the object frame by frame, 

Figure 9 shows a flow diagram of the streamer shown in 
Figure 2 , 

Figure 10 shows a block schematic diagram for combining the 
interactive content data with the video programme signal, 

Figure 11 shows the structure of a data packet used in this 
invention, and 

Figure 12 shows in block schematic form the manner of 
extracting the interactive content data from the video programme 
signal. 

In the Figures like reference numerals denote like parts. 

The interactive system shown in Figure 1 has apparatus 200 
for producing a data sequence that is representative of 
interactive content data associated with at least one object 
which is multiplexed 1080 with video and audio data 
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representative of the digital video programme. In the described 
embodiment, a data transport stream 1001 is applied to head end 
apparatus 10 of a satellite broadcast device 20 that transmits to 
a satellite 25 that / in turn! re-transmits the broadcast signal 
to plural users/ viewers 3 0 each having a respect broadcast 
receiving dish 31. The received signal may be applied to a PC 40 
having a TV card for interaction by a viewer. The received 
broadcast signal may also, or alternatively, be applied to a set 
top box 50 of a digital television 55 or a television with 
integrated set top box electronics. The set top box may be 
provided with a keyboard (not shown) or a mouse 56 for a viewer 
to manipulate an icon on the TV to select objects and interact 
with menus and operations that may be provided* The PC 40 may 
similarly be provided with a keyboard, but, as is customary, also 
a mouse so that the manner of use is the same as the set top box, 
so a viewer /user is able to select an object and perform 
interactive communication. Input and output to and from the PC 
is via a modem 45 to a public telephone network 60 which may be, 
for example, PSTN, ISTN, xDSL, or satellite, and the set top box 
50 is similarly connected to the network 60. The network 60 
interconnects the multiple viewers with an e-commerce management 
system 70 that may be a dedicated management system or a system 
inter-linked with an Internet service provider. In a system 
where a video programme is broadcast, the system 70 is connected 
to the broadcast providing system so that the system 7 0 can tie- 
in with the broadcast programme for maintaining a reference 
between the objects transmitted to a viewer. 

In the system of this invention an object which may be, for 
example, a person, physical objects such as clothing, a watch, 
cosmetics, musical instruments or, for example, a trademark has 
data associated with that object multiplexed (embedded) into the 
video programme signal of the programme that carries the object. 
To achieve this it is necessary to identify and track objects 
frame by frame' throughout the video programme. It is to be 
understood that although in the described embodiment the video 
programme is broadcast, the video programme could be on a digital 
video disk (DVD) , tape or any known means for storing a video 
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programme. The viewer upon selecting an object is then able to 
interact with details concerning the object. For example, where 
the object is a musician in a pop musical video, information may- 
be derived as to where the music record, clothing worn and 
advertised by the musician may be secured over the Internet. 

The first stage is to produce the interactive data that will 
be dynamically associated with the, or each, object in every 
frame of a programme in which the object appears- A five-minute 
video sequence, for example, will typically consist of 7,500 
frames, whereas a ninety-minute movie may be 135,000 frames. 

If the input video programme is not in a digital format, the 
programme must first be digitised by means known per se . 

Referring to Figure 2, the apparatus 2 00 for generating the 
interactive content data associated with an object in relevant 
frames of a programme is shown. The digitised programme from a 
digital video source 201 is divided into related shots 3 00 (shown 
in Figure 3) by a parser 400, shown in detail in Figure 4. In 
the context of this invention a "shot" is a single camera "take 11 
of a scene • A five-minute video sequence may typically have one 
hundred such shots or edits consisting of a series of frames Fn 
where, for example, Fn - 25 x 60 x 5 - 7,500 frames, whereas a 
ninety-minute video may have thousands of shots. If the 
digitised video programme is supplied with an optional edit list 
202, which edit list indicates at which frames the shots 3 00 
change, this may be utilised to divide the programme into the 
separate shots 300. 

Basically, the parser 400 deconstructs the video into a 
group of sequences 321, 322, 323 (Figure 3) . The sequences 
consist of a series of semantically related shots and, for 
example, one sequence may contain all the shots that feature the 
lead singer in a pop music video. Therefore, the function of the 
parser 400 is to deconstruct the programme into sequences unified 
by a common thread. The operation is necessary so that the 
tracker 800, described hereinafter, will only search for objects 
in sequences where they are likely to be found. The parser 400 
detects shot changes, camera angle changes, wipes, dissolves and 
any other possible editing function or optical transition effect. 
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The parser 4 00 shown in Figure 4 receives the digital programme 
and the end of a shot is detected 410, e.g. by comparing edge 
maps of each successive frame of the video programme and 
stipulating that an end of shot occurs when a change in location 
of the edge map occurs which exceeds a predetermined threshold. 
The criteria 420 to be used to determine the end of a shot is 
input into the cut/ shot detection programme by a user who is 
embedding data associated with an object into the video programme 
sequence* Information of different shots is put into an edit 
list 430. 

A number of frames are then selected in a key-frame 
identifier 440 from each shot 300 to become key-frames 500 (see 
also Figure 5) Which are representative of that shot 3 00. More 
than one key-frame may be needed for each shot where the shot 3 00 
includes, for example, complex camera moves, such as pans or 
zooms, so that one key-frame 500 is not representative of the 
total content. Furthermore, if the video programme is of a pop 
group, and the sequence starts with a long shot of all the band 
members and speedily zooms onto the lead singer and ends with the 
lead singer's face filling the screen, no single frame would be 
representative of the whole shot, but a valid selection of three 
key-frames would be, for example, the first frame 311, a frame 
312 about half-way through the zoom, and a final frame 313 (shown 
in Figure 3), : Thus, key-frames 311, 312 and 313 are 
automatically selected which are representative of the video 
content of the shot 3 00. 

As shown in Figures 3 and 4, the shots 300 are grouped into 
sequences by a scene grouper 450 which compares the key-frames 
311 - 313 from each shot 300 with the key-frames 311 - 313 from 
each others shot 304, 307. This is performed by comparing the 
key-frames from the shots using low level features such as colour 
correlelograms, data maps and textures. Shots that have similar 
content are grouped together into a hierarchical structure by the 
scene grouper 450 into groups of shots having a common theme. 
For example, in a pop music video, it may be that there are 
several different sets used, but one set may appear in many 
places in the video. The scene grouper 450 groups sequences of 
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the shots 300, 304 , 307 using the same set on one level and 
similar types of shots/ sequences of the same set at another 
level. In this way, a hierarchical structure, termed a content 
tree 460 , of sequences is built up. The purpose of the grouping 
is to aid in the selection of objects to be identified by 
interactive content data and also improve the efficiency of the 
subsequent tracking of the selected object through the video 
programme (described hereinafter) by ensuring that searching for 
a particular object is carried out only within related shots 300, 
304, 307 and not through all shots of the film. The parser 400 
thus assists the user to grasp the full structure and complexity 
of the video programme by providing a powerful browsing and 
object selection device as well as increasing the efficiency of 
the tracker by limiting tracking of an object to related shots, 
i.e. shots in sequences 321, 322, 323* 

Having grouped the shots 300 into sequences 321, 322, 323, 
sequence key-frames are selected from the key-frames 311, 312, 
313 of each shot to represent the sequence. A user wishing to 
input interactive content data representative of an object into a 
video programme may then use these high level key-frames to 
select those sequences of shots which contain objects of interest 
to the user. These key-frames are preferably presented to the 
user in a form representing the hierarchical structure in the 
content tree 460 of the sequences 321, 322, 323. An output 470 
of the scene grouper 450 is a number of sequences of single 
shots, key-frame 311, 312, 313 representing the sequences and a 
content tree showing the hierarchical relationship between the 
sequences, as reflected by the key-frames. 

The user intending to insert the interactive content data 
into the video programme views the hierarchical structure of the 
key frames and selects a first key-frame 311, as shown in Figure 
5, In a preferred embodiment, all the key-frames may be 
presented to a user on a screen in miniaturised form and the user 
may position a cursor over the miniaturised key frame and select 
that key-frame. A full-sized version of the key-frame may then 
be presented to the user for selection of objects from the key 
frame 311. The user then marks with a pointing device, such as a 
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mouse, an object 600 within the key-frame 311 which the user 
intends to associate with interactive content data embedded in 
the programme video (as shown in Figure 6) . The object may be 
marked by drawing a boundary box 610 around the object. To 
select the object 600 in the key-frame 311, the user clicks a 
mouse button when the cursor is at the top left corner and drags 
the mouse cursor to the bottom right corner of the object 6 00 so 
that the boundary box 610 is displayed around the selected object 
600. 

For example , to embed data information about a pop group 
tour date, the entire key-frame may be selected. If the key- 
frame contains a keyboard then the keyboard may be selected to 
advertise the keyboard and/or sell the keyboard on behalf of the 
keyboard manufacturer* Also, the lead singer who appears in the 
key-frame may also be selected. The boundary box shown in Figure 
6 is rectangular, which is a preferred default shape, but other 
shapes may be used such as a parallelogram or a user defined 
polygon. 

The selection of objects is made and the object identified 
600 , as shown in detail in Figure 7. Thus, the user- identifies 
objects 710, points to and clicks on the object 600 to provide 
initial object choices 715. As each object 600 is selected in 
the key-frame 311, attributes used to track the object through 
successive frames are calculated and compared with the attributes 
of objects already recorded 72 0 to ensure that the new object is 
distinctly different from all other objects already recorded for 
that frame. These attributes may include any of shape, size, 
position, colour, texture and intensity gradient of the object, 
as well as time series statistics based on these attributes. If 
a new object is too similar to previously recorded objects, the 
user is prompted for extra information about the new object. 
Otherwise, the attributes of the object are recorded. 

The selected object in block 725 is viewed isolated from the 
rest of the frame. The user may then change the boundary box 610 
to define the object 600 by discriminating 730 against other 
objects more precisely, or if two objects overlap so that they 
occupy the same location on the screen, the user may indicate 
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which object takes precedence by assigning a rank to each of the 
overlapping objects. For instance, in the example given above, 
information on the group's tour dates, which is associated with a 
whole frame, may be given a low rank so that f for example, any 
other object appearing anywhere in the frame will always have a 
higher rank and not be overridden by the data associated with the 
whole frame 311. This process is repeated for each of the key- 
frames 311 representing each of the sequences 321, 322, 323. 

As each object is selected in the key-frame, the next step 
is to identify the object using data and embed the date with the 
object. Preferably, record addresses of data are held in a 
database , the data being associated with a particular object or, 
alternatively, instead of using a record address , the data itself 
may be embedded. Preferably, a graphical user interface 750 is 
used to drag an icon representing the data onto the object 600 
within the frame 311. 

Thereby the user adds the advertising content to each object 
in the segmented frame using a "drop and drag" technique so that, 
for example, an icon representing the advertiser is dragged over 
the object using a mouse and the relevant data is automatically 
embedded into the object. This process continues until all 
objects have been embedded with interactive data. Thereby, data 
representative of an object is embedded 760 into the video 
programme signal to provide interactive content data associated 
with objects 765 and a number of key-frames associated with 
respective embedded content data as an output 77 0. 

Thus, the identifier 700 identifies the objects to have 
content embedded in them by accessing a small number of key- 
frames from each sequence and embedding the content. 

Having embedded object descriptors in key-frames and 
provided content it is necessary to track the objects through the 
successive frames of the video programme. 

Referring to Figure 8, it is necessary to track an object 
throughout the video programme and also as an object moves within 
frames and is occasionally obscured by other objects or leaves 
the frame being viewed, altogether. Basically, the objects are 
defined as a series of boundary shapes plus low-level feature 
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functions, e.g. shapes, edges, colour, texture and intensity 
gradient information. Using this representation of the objects, 
they are tracked through the remaining frames of the video 
sequence in an iterative fashion. When the plural objects have 
been tracked and located in every frame in which they appear, 
then the relevant content that was embedded in the first key- 
frame 311 is added automatically to the remaining frames of all 
sequences and this is the function of the object tracker 8 00, 
shown particularly in Figure 8. 

Uncut sequences and selected objects 810 are converted 815 
to a low-level representation 820 used to compare objects within 
a frame. For all frames, a distance measure is utilised to 
locate each object within each frame* A convenient distance 
measure is the Hausdorff measure, known per se / but this measure 
may be augmented with other techniques. Tracking 8 25 of the 
objects through sequential frames is iteratively provided whereby 
the object is initially defined in the key-frame as a two- 
dimensional geometric shape obtained by performing edge detection 
and segmenting out the edges encircled within the bounding box 
610. The object 610 is then located in the next frame 312 and 
the attributes of the object updated to reflect the changes in 
position and shape that have occurred between the frames. The 
object with these new attributes is then located in the next 
frame and the process of tracker 800 continues. 

Once the position of each object within all the frames of a 
sequence of shots has been determined, post-processing of the 
positions to smooth over occlusions and exits and entrances of 
objects is carried out. 

The system is impervious to lighting changes, occlusion, 
camera moves, shots , breaks and optical transition effects such 
as wipes, fades and dissolves. The system uses a variety of 
known techniques to enable automatic tracking in all types of 
vision environments, e.g. using a group of algorithms, the 
selection of which is dependent upon the visual complexity of the 
sequence. These algorithms are known per se, although the person 
skilled in the art may use heuristics to optimise performance for 
tracking. The data added to the objects in the key-frames is 
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then automatically added to the object in all frames as the 
object is tracked throughout the entire video sequence 830, 

A user may review the tracks produced and enter any 
corrections 835. The corrections are made by stopping the 
reviewed sequence at the erroneous frame, clicking on the object 
which is in error and dragging it to its correct position. Thus, 
using a graphical user interface, the video is stopped at the 
location in which the location of the object is incorrectly 
identified and the bounding box 610 is dragged and dropped at its 
correct location, thereby re-defining the attributes of the 
object for that frame and basing the definition of the object for 
subsequent frames on that new definition, thereby producing 
verified tracks 84 5. 

Finally, all frames in all sequences of the video will have 
relevant objects identified and embedded with interactive content 
data 850. 

Output from the tracker 800 is applied to a streamer 900, 
shown in Figure 9, in which the validity of the embedded 
interactive content data is checked, the order that the embedded 
interactive content data is output is synchronised, where 
necessary, with the audio/visual frames. 

The streamer checks that all objects in all frames have 
embedded content data 850 and that the content is labelled and 
valid using encoder setting 920 to act upon encoder and error 
checker 910. Verification 940 that the content is correctly 
labelled and valid occurs and the output 93 0 may be either a 
complete broadcasting compliant transport stream, such as MPEG- 
2/DVB audio, video and embedded objects and content data, or as 
embedded objects and content data alone* 

The streamer 900 must determine in which of three categories 
the embedded content data falls, namely frame-synchronous data, 
segment-synchronous data, or data to be streamed just- in-time. 
Frame synchronous data consists of the object positions, shapes, 
ranks and pointers to a table of pointers to data may be 
associated with the correct frame number in the video programme 
from source 201. Segment-synchronous data is used to update the 
table of pointers to embedded content data so that when objects 
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change, the embedded data changes. This data may be associated 
with the frame number at which the content changes. Data to be 
streamed "just in time" must be streamed to the end user before 
it is required by any of the objects. This transport stream is 
then packetised into MPEG- 2 /DVB compliant packets. 

If a fully embedded audio visual programme is required, the 
packetised transport stream and the video programme are 
multiplexed together, as shown in Figure 10. 

Referring to Figure 10, the different elements that 
constitute the embedded video programme are combined into a 
single transport stream 1001 in preparation for broadcasting by a 
network operator. The programme consists of a video stream 1010, 
an audio stream 1020, both of which streams are uncompressed. 
Both the video data 1010 and the audio data 1020 are encoded and 
compressed in respective MPEG-2 elementary encoders 1015 and 1025 
to produce elementary streams of data 103 0, 1035 respectively* 
MPEG-2 compliant data sequence 93 0 is error checked 1037 to 
produce an elementary stream of data 1040. The elementary 
streams 1030, 1035 and 1040 are applied to packetisers 1050, 1055 
and 1060, which each accumulate data into fixed size blocks to 
which is added a protocol header. The output from the 
packetisers is termed a packetised elementary stream (PES) 1070. 
The packetised elementary streams 1070, in combination with 
digital control data (PSI) 1075, is applied to a systems layer 
multiplexer 1080 having a systems clock 1085. The PES packet is 
a mechanism to convert continuous elementary streams of 
information 1030, 1035 and data sequence 930 into a stream of 
packets. Once embedded in PES packets the elementary streams may 
be synchronised with time stamps. This is necessary to enable 
the receiver (PC or TV) to determine the relationship between all 
the video, audio and data streams that constitute the embedded 
video programme. 

Each PES packet is fed to the system multiplexer 1030. There 
the packets are encapsulated into transport packets to form the 
transport stream 1001 that is used for broadcast. In this 
respect, the transport stream 1001 carries packets in 188 byte 
blocks and the transport stream 1001 constitutes a full so-called 
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eMUSJE channel that is fed to the network operator for broadcast. 
In essence, the transport stream is a general purpose way of 
combining multiple streams using fixed length packets. 

The structure of a packet is shown in Figure 11. The packet 
1100 shown in Figure 11 has a header 1110 with a synchronisation 
byte, a 13-bit packet identifier (PID) and a set of flags to 
indicate how the packet should be processed. The transport 
multiplexer assigns a different packet identifier to each PES 
1070 to uniquely identify the individual streams. In this way, 
the packetised data sequence 930 is uniquely identified. The 
synchronisation of the elementary streams is facilitated by 
sending time stamps in the transport stream 1001. 

Two types of time stamps may be used: 

1* A reference time stamp to indicate the current time, 

that is clock 1085 information, and 
2, A decoding time stamp. 

The decoding time stamps are inserted into the PES to 
indicate the exact time when the data stream has to be 
synchronised with the video and audio streams. The decoding time 
stamp relies on the reference time stamp for operation. After 
the transport stream has been broadcast, the PC or TV uses the 
time stamps to process the data sequence in relation to the video 
and audio streams. 

In order for the receiver (PC or TV) to know how to 
decode the channel, it needs to access a set of signalling tables 
known as Programme Specific Information (PSI) labels which are 
sent as separate packets within the transport stream 1001 with 
their own PID tables. There are two tables that are needed to 
enable the receiver to decode a channel* The first is the 
programme association table (PAT) 1130 which lists all the 
channels that are available within the satellite broadcast and 
has a packet ID (PID) value of o which makes it easy to identify. 
In the example, the eMUSE channel, i.e. the channel carrying the 
video programme, is represented as PID 111. 

A programme table map (PMT) 1140 identifies all the 
elementary streams contained in the embedded video signal. Each 
elementary stream is identified by a PID value, e.g. video from 
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video camera 1 is PID 71. The data sequence 930 has a PID value 
92 in the example of Figure 11. The receiver video and audio 
decoders search the PMT table to find the appropriate packets to 
decode* Similarly, the programme for retrieving the embedded 
data searches the PMT to find the data sequence which, in the 
example of Figure 11, is PID 92 « The data retrieval programme 
then filters out these packets and synchronises them with the 
appropriate video and audio to enable the user to select the 
various objects. 

Having embedded the interactive content data into the video 
programme signal, it is broadcast and the manner of reception and 
retrieval of the data will now be explained with reference to 
Figure 12. 

Hardware is provided on a satellite receiver card 1210 which 
resides on the user's PC 40 or digital set top box 50 and 
software allows the viewer to interact with the dynamic objects 
in the broadcast, for example to facilitate Internet access and 
Internet browsers, such as Internet Explorer and Netscape and, 
for TV applications, is compatible with Sun's Open TV operating 
system* 

The received MPEG-2/DVB signal is separated into MPEG-2 
video 1215, MPEG-2 audio 122 0 and the data sequence 930 and the 
decoded video 122 5, audio and data sequence is applied to a 
synchroniser 1230. Output from the synchroniser comprising the 
video programme' with embedded interactive content data is 
displayed 1240 by the PC VDU or TV screen* 

A user clicks a mouse 56 or presses a remote control button 
at a frame containing an object of interest, which causes the 
display on the screen to split in two. For example, on the left 
hand screen, the video programme continues to run as normal and, 
on the right hand screen, the objects present in the frame which 
was active the time the mouse was clicked, are displayed as cut- 
outs, with the intervening spaces blanked out. The user then 
clicks on the object of interest to see which advertisers it 
represents, e.g. if the user clicks on the lead singer, then the 
screen will display the lead singer only and a textual list of 
advertisers or an icon-based display of advertisers will be 
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viewed. If the user clicks on the advertiser's name or icon, the 
user goes directly to view the advertised products. 

After interacting with the site the user may decide to 
purchase the product via an e-commerce transaction. Further, if 
the user clicks on the suit of the lead singer, the entire 
catalogue of the suit manufacturer may be made available as part 
of the streamed digital broadcast. This return path via the 
Internet is purely to facilitate a transaction as the data 
sequence 93 0 initiates the push technology approach to streaming 
advertising information once the user has selected amongst the 
numerous objects within the frame. 

Although the user can interact with the broadcast in such an 
on-line manner as described above, alternatively, the data may be 
viewed off-line, i.e. while a viewer continues to watch a 
programme, the user may select various frames during the 
broadcast and store the frames for later retrieval of the 
associated data. Where there is not sufficient local memory to 
store the data, addresses of the data in local or remote 
databases, e.g. Web sites, are stored and the end user is able to 
subsequently access the databases to retrieve the data. The user 
then selects with the mouse or the remote control the object 600 
of interest and another screen may then be displayed showing the 
object 600 and a menu of data elements associated with that 
object. The user clicks one of the menu items and is able to 
directly view data on the advertised product or be given access 
to a Web site over the Internet. Alternatively, as soon as a 
user selects a menu item, a catalogue may be viewed which has 
been embedded in the broadcast signal. 

The data which the end user accesses may be streamed with a 
broadcast signal or may be held in a local data base which may be 
pre-loaded into the end user's device prior to viewing the video 
sequence. When viewing information streamed with a broadcast, 
the information associated with a particular programme is 
streamed in parallel with the programme and stored locally. When 
the user selects an object, this local data is viewed. 
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Claims: 

1* An apparatus for associating data representative of 
an object with a digital video programme including means 
for providing a digital video programme having plural 
individual frames at least some of which incorporate said 
object, means for parsing the video programme by 
identifying separate shots in the video programme to 
produce an edit list, means for identifying shots 
containing related content to form a sequence of shots 
containing related content, means for selecting at least 
one key-frame from within each sequence of shots, means 
for selecting said object within the key-frame with which 
data is to be associated, means for extracting attributes 
of the object from the key-frame, means for associating 
interactive data with the object in the key-frame, 
tracking means for utilising the attributes of the object 
for tracking the object through the sequence of shots, 
whereby said interactive data is associated with the 
object in the sequence of shots and said interactive 
content data is embedded with data representative of said 
object in a data sequence. 

2* An apparatus as claimed in claim 1, wherein the 
means for identifying shots containing related content to 
form a sequence of shots containing related content 
includes means for producing a hierarchy of groups of 
shots , 

3. An apparatus as claimed in claims 1 or 2, wherein 
said means for parsing include means for inputting 
criteria to be used to recognise a change of shot. 

4* An apparatus as claimed in any of claims 1 to 3, 
wherein the means for extracting attributes of the object 
includes means for isolating the object within a boundary 
formed on the frame , means for performing edge detection 
within the boundary to identify and locate edges of said 
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object, and storing means for storing a geometric model 
of said object ♦ 

5. An apparatus as claimed in any of claims 1 to 4, 
wherein said means for extracting attributes of said 
object also includes means for recording at least one of 
the attributes of shape, size, position, colour, texture, 
intensity gradient of said object, and time series 
statistics based on said attributes. 

6. An apparatus as claimed in any of the preceding 
claims, wherein said means for extracting attributes of 
said object includes means for comparing said attributes 
of said object with attributes of objects previously 
stored to determine whether the object is distinguishable 
therefrom, and when said object is determined not to be 
distinguishable, providing means for re-defining the 
object. 

7. An apparatus as claimed in any of the preceding 
claims, wherein said means for extracting attributes of 
said object includes means for comparing the location in 
the frame of said object with the location of objects 
already stored for that frame to determine whether that 
object is distinguishable therefrom, and where the 
location of said object is not distinguishable from the 
location of another object providing means for assigning 
rank to the objects to determine which object will be 
associated with that location* 

8. An apparatus, as claimed in any of the preceding 
claims, wherein the means for utilising the attributes of 
the object for tracking the object includes means for 
updating the stored attributes of the object as the 
attributes of the object change from frame to frame. 

9. An apparatus as claimed in any of the preceding 
claims, wherein said tracking means utilising the 
attributes of the object for tracking the object includes 
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plural algorithm means for calculation of independent 
tracks of objects for use depending on the visual 
complexity of a sequence to automatically track said 
objects in different types of visual environment* 

10* An apparatus as claimed in any of the preceding 
claims, wherein said tracking means for utilising the 
attributes of the object for tracking the object includes 
means for converting all the frames to be tracked to a 
low-level representation, means for determining the 
position of each object in the frames by minimising a 
distance measure to locate each object in each frame, 
means for processing the positions of said object to 
smooth over occlusions and the entrances and exits of 
objects into and out of said frames, and means for 
reviewing the object within a tracked sequence and for 
correcting the location attributes of any misplaced 
objects . 

11. An apparatus, as claimed in any of the preceding 
claims, wherein the means for associating interactive 
data with the object in the key-frame includes means for 
providing a database of different types of data including 
one or more of URLs , HTML pages, video clips, audio 
clips, text files and multimedia catalogues, and means 
for selecting said interactive content data from the 
database to associate with said object. 

12. An apparatus, as claimed in any of the preceding 
claims, wherein the means for associating interactive 
data with the object in the key-frame produces said data 
sequence using means for determining whether the embedded 
interactive content data is frame synchronous data 
associated with object positions, shapes, ranks and 
pointers in a frame, or group-synchronous data associated 
with all the objects in a group, or is data to be 
streamed just in time, wherein means are provided for 
associating frame synchronous data with the corresponding 
frame, means are provided for associating group 
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synchronous data with the frame at which a group changes, 
and means are provided for streaming just in time data to 
a user before it is required to be associated with the 
corresponding objects. 

13. An apparatus as claimed in any of the preceding 
claims, wherein means are provided to associate different 
interactive content data with respectively different 

ob j ects . 

14. A method for associating interactive data 
representative of an object with a digital video 
programme including the steps of: 

a) providing a digital video programme having a plurality 
of individual frames at least some of which incorporate 
said object with which data is to be associated, 

b) parsing the video programme by identifying separate 
shots in the video programme to produce an edit list, 

c) identifying, from the edit list, shots containing 
related content to form a sequence of shots containing 
related content, 

d) selecting at least one key-frame containing the object 
from within the sequence of shots, 

e) locating said object within the at least one key- 
frame, 

f) extracting attributes of the object from the at least 
one key-frame, 

g) associating interactive data with the object in the at 
least one key-frame, 

h) tracking the object through the sequence of shots 
utilising the attributes of the object, 

i) associating said interactive data with the object in 
frames in the sequence of shots , and 

j) embedding said interactive data with data 
representative of said object in a data sequence 
representative of the digital video programme* 
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15. A method as claimed in claim 14 , wherein step b) 
includes the step of inputting criteria to be used to 
recognise a change of shot, 

16. A method as claimed in claims 14 or 15, wherein step 
c) includes the step of producing a hierarchy of groups 
of sequences of shots* 

17. A method as claimed in any of claims 14 to 16, 
wherein step e) includes the steps of: isolating the 
object within a boundary formed on the frame, performing 
edge detection within the boundary to identify and locate 
edges of said object, and step f) includes storing a 
geometric model of said object. 

18. A method as claimed in any of claims 14 to 17, 
wherein step f) includes the step of recording at least 
one of the attributes of shape, size, position, colour, 
texture, intensity gradient of said object, and time 
series statistics based on said attributes. 

19. A method as claimed in any of claims 14 to 18, 
wherein step f) includes the step of comparing said 
attributes of said object with attributes of objects 
previously stored to determine whether the object is 
distinguishable therefrom, and when said object is 
determined not to be distinguishable, the step of re- 
defining the object. 

20. A method as claimed in any of claims 14 to 19, 
wherein step f ) includes the step of comparing the 
location in the frame of said object with the location of 
objects already stored for that frame to determine 
whether that object is distinguishable therefrom, and 
where the location of said object is not distinguishable 
from the location of another object, the step of 
assigning rank to the objects to determine which object 
will be associated with that location. 
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21. A method, as claimed in any of claims 14 to 20, 
wherein step h) includes the step of updating the stored 
attributes of the object as the attributes of the object 
change from frame to frame, 

22. A method as claimed in any of claims 14 to 21, 
wherein step h) includes the step of using plural 
algorithm means for calculation of independent tracks of 
objects for use depending on the visual complexity of a 
sequence automatically to track said objects in different 
types of visual environment. 

23. A method as claimed in any of claims 14 to 22, 
wherein step h) includes the steps of converting all the 
frames to be tracked to a low-level representation, 
determining the position of each object in the frames by 
minimising a distance measure to locate each object in 
each frame, processing the positions of said object to 
smooth over occlusions and the entrances and exits of 
objects into and out of said frames, reviewing the object 
within a tracked sequence and correcting the location 
attributes of any misplaced objects. 

24. A method, as claimed in any of claims 14 to 23, 
wherein step g) includes the steps of providing a 
database of different types of data including one or more 
of URLs, HTML pages, video clips, audio clips, text files 
and multimedia catalogues, and selecting said interactive 
content data from the database to associate with said 
object . 

25. A method, as claimed in any of claims 14 to 24 , 
wherein step j) includes determining whether the embedded 
interactive content data is frame synchronous data 
associated with object positions, shapes, ranks and 
pointers in a frame, or group-synchronous data associated 
with all the objects in a group, or is data to be 
streamed just in time, and associating frame synchronous 
data with the corresponding frame, associating group 
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synchronous data with the frame at which a group changes, 
and streaming just in time data to a user before it is 
required to be associated with the corresponding objects, 
respectively. 

26. A method as claimed in any of claims 14 to 25, 
wherein in steps d) to j) different interactive content 
data are associated with respectively different objects. 

27. A computer program comprising code means for 
performing all the steps of the method of any of claims 
14 to 26 when the program is run on one or more 
computers . 

28. A computer program as claimed in claim 27, wherein 
the computer program is embodied on a computer-readable 
medium. 
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COMBINED DECLARATION AND POWER OF ATTORNEY 



(ORIGINAL, DESIGN, NATIONAL STAGE OF PCT, SUPPLEMENTAL, DIVISIONAL, 
CONTINUATION, OR C-I-P) 

As a below named inventor, I hereby declare that: 



TYPE OF DECLARATION 



This declaration is of the following type: 



(check one applicable item below) 

[ ] original. 
[ ] design. 

NOTE: With the exception of a supplemental oath or declaration submitted in a reissue, a supplemental oath or 

declaration is not treated as an amendment under 37 CFR 1.312 (Amendments after allowance). M.P.KP. Section 
714.16, 7 th Ed. 



[ ] supplemental. 



NOTE: If the declaration is for an International Application being filed as a divisional continuation or continuation-in- 
part application, do not check next item; check appropriate one of last three items. 



[x] national stage of PCT. 



NOTE: If one of the following 3 items apply, then complete and also attach ADDED PAGES FOR DIVISIONAL, 
CONTINUATION OR C-I-P. 

NOTE: See 37 C.F.R. Section 1 63(d) (continued prosecution application) for use of a prior nonprovisional application 

declaration in the continuation or divisional application being filed on behalf of the same or fewer of the inventors 
named in the prior application. 

[ ] divisional. 
[ ] continuation. 



NOTE: Where an application discloses and claims subject matter not disclosed in the prior application, or a continuation 
or divisional application names an inventor not named in the prior application, a continuation-in-part application 
must be filed under 37 CF.R. Section 1.53(b) (application filing requirements-nonprovisional application). 



[ ] continuation-in-part (C-I-P). 



(Declaration and Power of Attorney— page 1 of 8) 1-1 



INVENTORSHIP IDENTIFICATION 



WARNING: If the inventors are each not the inventors of all the claims, an explanation of the facts, including the 

ownership of all the claims at the time the last claimed invention was made, should be submitted. 

My residence, post office address and citizenship are as stated below, next to my name. I believe that I 
am the original, first and sole inventor (if only one name is listed below) or an original, first and joint 
inventor (if plural names are listed below) of the subject matter that is claimed, and for which a patent 
is sought on the invention entitled: 

TITLE OF INVENTION 

INTERACTIVE SYSTEM 



SPECIFICATION IDENTIFICATION 

The specification of which: 

(complete (a), (b), or (c)) 

(a) [ ] is attached hereto. 

NOTE: "The following combinations of information supplied in an oath or declaration filed on the application filing date 
with a specification are acceptable as minimums for identifying a specification and compliance with any one of the 
items below will be accepted as complying with the identification requirement of 37 C.F.R. Section 1,63: 

"(1) name of inventor (s), and reference to an attached specification which is both attached to the oath or 
declaration at the time of execution and submitted with the oath or declaration on filing; 

"(2) name of inventor(s), and attorney docket number which was on the specification as filed; or 

"(3) name of inventor (s), and title which was on the specification as filed. " 

Notice of July 13, 1995 (1177 O.G. 60). 

(b) [ ] was filed on , [ ] as Application No. 

[ ] and was amended on (if applicable). 

NOTE: Amendments filed after the original papers are deposited with the PTO that contain new matter are not accorded a 
filing date by being referred to in the declaration. Accordingly, the amendments involved are those filed with the 
application papers or, in the case of a supplemental declaration, are those amendments claiming matter not 
encompassed in the original statement of invention or claims. See 37 C.F.R. Section 1.67. 

NOTE: "The following combinations of information supplied in an oath or declaration filed after the filing date are 

acceptable as minimums for identifying a specification and compliance with any one of the items below will be 
accepted as complying with the identification requirement of 37 C.F.R. Section 1.63: 

(A) application number (consisting of the series code and the serial number, e.g., 08/123,456); 

(B) serial number and filing date; 

(C) attorney docket number which was on the specification as filed; 

(D) title which was on the specification as filed and reference to an attached specification which is 
both attached to the oath or declaration at the time of execution and submitted with the oath or 
declaration; or 

(E) title which was on the specification as filed and accompanied by a cover letter accurately 
identifying the application for which it was intended by either the application number (consisting of the 
series code and the serial number, e.g., 08/123,456), or serial number and filing date. Absent any 
statement(s) to the contrary, it will be presumed that the application filed in the PTO is the application 
which the inventor(s) executed by signing the oath or declaration. 

M.P.E.P. Section 601.01(a), 7th ed. 
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(c) [x] was described and claimed in PCT International Application No. PCT/IB00/00135 
filed on January 28, 2000 and as amended under PCT Article 19 on (if any). 



SUPPLEMENTAL DECLARATION (37 C.F.R. Section 1.67(b)) 

(complete the following where a supplemental declaration is being submitted) 

[ ] I hereby declare that the subject matter of the 

[ ] attached amendment 

[ ] amendment filed on . 

was part of my/our invention and was invented before the filing date of the original 
application, above identified, for such invention. 

ACKNOWLEDGMENT OF REVIEW OF PAPERS AND DUTY OF CANDOR 

I hereby state that I have reviewed and understand the contents of the above-identified 
specification, including the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose information, which is material to patentability as defined in 
37, Code of Federal Regulations, Section 1.56, 

(also check the following items, if desired) 

[ ] and which is material to the examination of this application, namely, information 

where there is a substantial likelihood that a reasonable Examiner would consider it 
important in deciding whether to allow the application to issue as a patent, and 

[ ] in compliance with this duty, there is attached an information disclosure 
statement, in accordance with 37 C.F.R. Section L98. 

PRIORITY CLAIM (35 U.S.C. Section 119(a)-(d)) 

NOTE: "The claim to priority need be in no special form and may be made by the attorney or agent if the foreign 

application is referred to in the oath or declaration as required by Section J. 63. The claim for priority and the 
certified copy of the foreign application specified in 35 U.S.C. Section 119(b) must be filed in the case of an 
interference (Section 1.630), when necessary to overcome the date of a reference relied upon by the examiner, when 
specifically required by the examiner, and in all other situations, before the patent is granted. If the claim for 
priority or the certified copy of the foreign application is filed after the date the issue fee is paid, it must be 
accompanied by a petition requesting entry and by the fee set forth in Section L17(i). If the certified copy is not in 
the English language, a translation need not be filed except in the case of interference; or when necessary to 
overcome the date of a reference relied upon by the examiner; or when specifically required by the examiner, in 
which event an English language translation must be filed together with a statement that the translation of the 
certified copy is accurate. "37 C.F.R. Section 1.55(a). 

I hereby claim foreign priority benefits under Title 35, United States Code, Section 1 19(a)-(d) 
of any foreign application(s) for patent or inventor's certificate or of any PCT international 
application(s) designating at least one country other than the United States of America listed below 
and have also identified below any foreign application(s) for patent or inventor's certificate or any 
PCT international application(s) designating at least one country other than the United States of 
America filed by me on the same subject matter having a filing date before that of the application(s) of 
which priority is claimed. 

(complete (d) or (e)) 
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(d) [ ] no such applications have been filed. 

(e) [x] such applications have been filed as follows. 

NOTE: Where item (c) is entered above and the International Application which designated the U.S. itself claimed priority 
check item (e), enter the details below and make the priority claim. 



PRIOR FOREIGN/PCT APPLICATION(S) FILED WITHIN 12 MONTHS 
(6 MONTHS FOR DESIGN) PRIOR TO THIS APPLICATION 
AND ANY PRIORITY CLAIMS UNDER 35 U.S.C. SECTION 119(a)-(d) 



COUNTRY (OR 
INDICATE IF 
PCT) 


APPLICATION NUMBER 


DATE OF FILING 
DAY, MONTH, YEAR 


PRIORITY 
CLAIMED 
UNDER 35 USC 
119 


GB 


9902235.0 


1 February 1999 


TxlYES [ ]NO 








r iyes r ino 








r iyes r ino 








f IYES [ ]NO 








r iyes r ino 



CLAIM FOR BENEFIT OF PRIOR U.S. PROVISIONAL APPLICATION(S) 

(35 U.S.C. Section 1 19(e)) 

I hereby claim the benefit under Title 35, United States Code, Section 119(e) of any United 
States provisional applications) listed below: 

PROVISIONAL APPLICATION NUMBER FILING DATE 

/ 

/_ 

/ 



CLAIM FOR BENEFIT OF EARLIER U.S./PCT APPLICATION(S) 
UNDER 35 U.S.C. SECTION 120 

[ ] The claim for the benefit of any such applications are set forth in the attached 

ADDED PAGES TO COMBINED DECLARATION AND POWER OF ATTORNEY 
FOR DIVISIONAL, CONTINUATION OR CONTINUATION-IN-PART (C-I-P) 
APPLICATION. 
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ALL FOREIGN APPLICATTON(S), IF ANY, FILED MORE THAN 12 MONTHS 
(6 MONTHS FOR DESIGN) PRIOR TO THIS U.S. APPLICATION 



NOTE: If the application filed more than 12 months from the filing date of this application is a PCT filing forming the 

basis for this application entering the United States as (J) the national stage, or (2) a continuation, divisional, or 
continuation-in-part, then also complete ADDED PAGES TO COMBINED DECLARATION AND POWER OF 
ATTORNEY FOR DIVISIONAL, CONTINUATION OR C-I-P APPLICATION for benefit of the prior US. or PCT 
application(s) under 35 U.S.C Section 120. 



POWER OF ATTORNEY 

I hereby appoint the following practitioner(s) to prosecute this application and transact all 
business in the Patent and Trademark Office connected therewith. 

(list name and registration number) 



JOSEPH H. HANDELMAN, 26179 
JOHN RICHARDS, 3 1053 
RICHARD J. STREIT, 25765 
PETER D. GALLOWAY, 27885 
IAIN C. BAILLIE, 24090 
RICHARD P. BERG, 28145 



JULIAN H. COHEN, 20302 
WILLIAM R. EVANS 25858 
JANET L CORD, 33778 
CLIFFORD J. MASS, 30086 
CYNTHIA R. MILLER, 34678 



(Check the following item, if applicable) 

[ ] I hereby appoint the practitioners) associated with the Customer Number provided 
below to prosecute this application and to transact all business in the Patent and 
Trademark Office connected therewith. 

[ ] Attached, as part of this declaration and power of attorney, is the authorization of the 
above-named practitioner(s) to accept and follow instructions from my 
representative(s). 

NOTE: "Special care should be taken in continuation or divisional applications to ensure that any change of 

correspondence address in a prior application is reflected in the continuation or divisional application. For 
example, where a copy of the oath or declaration from the prior application is submitted for a continuation or 
divisional application filed under 37 CFR 1.53(b) and the copy of the oath or declaration from the prior 
application designates an old correspondence address, the Office may not recognize, in the continuation or 
divisional application, the change of correspondence address made during the prosecution of the prior application. 
Applicant is required to identify the change of correspondence address in the continuation or divisional application 
to ensure that communications from the Office are mailed to the current correspondence address. 37 CFR 
1. 63(d)(4)." Section 601.03, M.P.E.P., 7th Ed 
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SEND CORRESPONDENCE TO 



DIRECT TELEPHONE CALLS TO: 



(Name and telephone number) 



Ladas & Parry 
26 West 61 st Street 
New York, N.Y. 10023 



William R. Evans 

(212) 708-1930 



(complete the following if applicable) 



Since this filing is a [ ] continuation [ ] divisional there is attached hereto a Change of 
Correspondence Address so that there will be no question as to where the PTO should direct all 
correspondence. 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the like so made are punishable by 
fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code, and that 
such willfiil false statements may jeopardize the validity of the application or any patent issued 
thereon. 



DECLARATION 



(Declaration and Power of Attorney— page 6 of 8) 1-1 



SIGNATURE(S) 



NOTE: Carefully indicate the family (or last) name, as it should appear on the filing receipt and all other document 

NOTE: Each inventor must be identified by full name, including the family name, and at least one given name without 
abbreviation together with any other given name or initial, and by his/her residence, post office address and 
country of citizenship. 37 C.F.R. Section 1.63(a)(3). 

NOTE: Inventors may execute separate declarations/oaths provided each declaration/oath sets forth all the inventors. 

Section 1.63(a)(3) requires that a declaration/oath, inter alia, identify each inventor and prohibits the execution of 
separate declarations/oaths which each sets forth only the name of the executing inventor. 62 Fed. Reg. 53,131, 
53,142, October 10, 1997, 

FulJname^^^e or first inventor 

r fek J, , RAINSFORD 



Patrivjv _ . 

yGNm^Namey~ (Middle Initial Name) n ^ Family (UrZast Name) 

Inventor's signature (x) 
Date M 0CT 2 2 2901 Country < 
Residence 80 Park Avenue. Sandvmount DubHn4. IRELAND 
Post Office Address SAME AS ABOVE 



(Middle Initial or lyamej a 
j^Zstn* ^^^^^ 
Countrv of Citizenshin lRISH 




Full name of second joint inventor, if any 

(Given Name) (Middle Initial or Name) Family (Or Last Name) 

Inventor's signature „ 

Date Country of Citizenship _ 

Residence _ 

Post Office Address 



Full name of third joint inventor, if any 

(Given Name) (Middle Initial or Name) Family (Or Last Name) 

Inventor's signature _ 

Date Country of Citizenship „ 

Residence 

Post Office Address _ 
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(check proper box(es)for any of the following added page(s) 
that form a part of this declaration) 

[ ] Signature for fourth and subsequent joint inventors. Number of pages added 

* * * 

[ ] Signature by administrator(trix), executor(trix) or legal representative for deceased or 
incapacitated inventor. Number of pages added 

* * * 

[ ] Signature for inventor who refuses to sign or cannot be reached by person authorized under 
37 C.F.R. Section 1.47. Number of pages added 

* * * 

[ ] Added page for signature by one joint inventor on behalf of deceased inventor(s) where legal 
representative cannot be appointed in time. (37 C.F.R. Section 1.47) 

* * * 

[ ] Added pages to combined declaration and power of attorney for divisional, continuation, or 
continuation-in-part (C-I-P) application. 

[ ] Number of pages added 

* * * 

[ ] Authorization of practitioner (s) to accept and follow instructions from representative. 



(If no further pages form a part of this Declaration, 
then end this Declaration with this page and check the following item) 



[x] This declaration ends with this page. 
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